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Essential Questions & Data Sources for Continuous Improvement of Undergraduate 
STEM Teaching and Learning is intended to be a useful resource for the individuals 
and institutions who work together to help foster continuous improvement of 
undergraduate STEM education. Informed by the broad literature on post-secondary 
STEM education, this resource draws primarily from the Association of American 
Universities’ experience in implementing the AAU Undergraduate STEM Education 
Initiative with our member campuses. It provides a set of questions that can be used at 
multiple levels within the university to assess progress along the set of key institutional 
elements identified in AAU’s Framework for Systemic Change in Undergraduate 
STEM Teaching and Learning. Questions are presented to examine the pedagogical, 
scaffolding, and cultural elements of the framework at each organizational level in 
the institution. Data sources and analytical tools available to answer these questions 
are profiled in a subsequent section. A final section provides guidance to address 
cross-cutting challenges to assessing improvement of undergraduate STEM teaching 


and learning. 


Through reflecting on these key questions, institutions can engage in deep discussions 
about undergraduate education; aggregate individual and program-level data to 
obtain an institutional portrait; and identify cross-cutting teaching and learning 
issues. Strong measures and indicators provide a foundation to advance larger cross- 
department, -college, -institution and potentially multi-institutional conversations 
among faculty members and institutional leaders, and to inform decision-making 
about reforms in undergraduate STEM education. 


Essential Questions identifies key questions institutions can use to: 


> engage institutional leaders and faculty members in discussions about teaching and 
learning; 


> establish strong measures of teaching effectiveness both to describe program 
performance and to incorporate in meaningful ways into faculty reward structures; 


> ensure quality and inclusive educational experiences for all students across all 
programs; 

> support the development or redesign of learning spaces; and 

> demonstrate the institution's commitment to the importance of evidence-based, 


student centered teaching to faculty members, students, and other relevant internal 
and external audiences. 


FRAMEWORK ELEMENTS 


The Framework for Systemic Change in Undergraduate STEM 
Teaching and Learning, developed as part of AAU's Undergraduate 
STEM Education Initiative, provides a set of key institutional 
elements that need to be addressed in order to bring about 
sustainable change. 
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ESSENTIAL QUESTIONS 


INSTITUTIONS 


> Which schools/colleges and departments at the institution make the most use of 
evidence-based pedagogical educational practices? How are they being rewarded? 
How are other departments encouraged to follow their lead? 


> What is being done across schools and departments to ensure that all students are 
succeeding? Are measures of student success disaggregated demographically? How 
are improvements reflected in data over time? 


>» What measures does the institution use to assess student progression/retention/ 
completion and other relevant factors? How often do top institutional leaders 
look at these data and discuss them with deans, department chairs, students, and 
others? 


>» How are needs to support the use of evidence-based pedagogy, as well as projected 
future enrollments and demographics, factored into facilities planning (e.g., 
learning spaces) at the institutional level? 


> What expectations have top institutional leaders articulated for each school/college 
and department to develop measures of evidence-based teaching beyond student 
evaluations? 


> How are measures of teaching excellence included in a meaningful way in annual/ 
merit review, promotion, and tenure processes for all instructors at the institution? 


UT 


SCHOOLS/COLLEGES 


> How has the school/college assured that all departments have made dedicated 
efforts to define core competencies and skills and to connect these outcomes to 
learning goals? 


> How much do departments within the school/college vary in the amount of time 
instructors are spending on various kinds of activities in the classroom? Is there a 
relationship between these activities and student success? 


> How well are students doing in their progression/retention/completion in STEM 
courses? How does this success rate compare across groups, over time, and 
with similar programs at peer institutions? How do the progression/retention/ 
completion rates for students broken out by relevant demographic categories vary 
between departments and with peer institutions? 


>» What resources support instructional improvements and what effects do these 
efforts have on reducing the gap in student achievement across demographic 
groups? What is the relationship between student achievement data and 
instructional practices, and how can these data further inform changes in 
instructional practices? 


> In what ways do deans use data on student progression/retention/completion to 
inform discussions with department chairs and instructors to facilitate program 
improvement? 


> How do deans make clear that they expect evidence-based pedagogy from 
potential new faculty hires? 


DEPARTMENTS 


> Do all of the courses in the department have articulated learning goals, and are 
these made clear to students? What process exists to ensure that individual course 
learning goals connect to learning goals for the program, major, and department? 


> What are the demographics of students in the department? What are the 
progression/retention/completion rates for students in the department or major 
broken out by relevant demographic categories? How do these compare with other 
departments and what steps are being taken to improve these rates? 


>» What actions has the department chair taken to encourage instructors to take 
advantage of both on-campus and off-campus (e.g., through relevant disciplinary 
societies) resources and professional development related to pedagogy? How 
many instructors have taken advantage of these resources and what notable 
improvements have occurred as the result? 


>» What resources are available to instructors in the department for encouraging all 
students to succeed, and what steps have been taken to ensure all instructors take 
advantage of these resources? 


> To what extent do departmental instructors have access to learning spaces that 
support evidence-based pedagogy? What training in the use of those facilities is 
available to instructors in the department? 


> What is the department chair’s and distinguished faculty members’ support of 
evidence-based pedagogy? How well-known is this support to instructors and 
students? 


> What are the biggest barriers to evidence-based pedagogy for instructors in the 
department and how is the chair working to address them? How often does the 
chair discuss these issues with the dean or other institutional leaders? 


> How are all faculty who participate in annual/merit, promotion, and tenure 
evaluations educated about the meaningful inclusion of measures of teaching 
excellence in those processes? How closely does the chair review the outcomes of 
those processes to ensure teaching is indeed meaningfully included? 


COURSE LEVEL 


> Are learning goals clearly stated on the syllabus? To what extent do students in the 
class understand the course learning goals? What steps has the instructor taken to 
ensure that class activities and assignments are linked to learning goals? 


> How much time does the instructor spend on various kinds of activities in the 
classroom? How are these activities assessed? 


> To what extent does the instructor understand the biases he/she may bring to the 
classroom, and what steps has the instructor taken to mitigate these to ensure that 
all students are succeeding? 


> To what extent does the instructor take advantage of both on-campus and off- 
campus resources and professional development related to pedagogy? 


> To what extent does the instructor participate in discussions about using data to 
help drive program improvement? To what extent are they aware of data about 
their students and courses? 


> Does the instructor believe that meaningful measures of teaching will factor into 
their own performance, promotion, and tenure reviews? If so, do they have a clear 
understanding of how teaching fits into the overall review process? How frequently 
are these measures discussed with peers and those who will be evaluating 
performance? 
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DATA SOURCES & ANALYTICAL TOOLS 


The following is a list of established and emerging data sources and analytical tools. 
This is a rapidly changing landscape, and new means are being developed to answer the 
questions we've identified. For a current list of analytical tools visit www.aau.edu/stem. 


INSTITUTIONAL DATA & VISUALIZATIONS: Institutions possess a wealth of data on 
students and their performance that may be used to help understand and improve 
teaching and learning. New visualization tools can help reveal patterns in data and 
may be useful for driving discussion and decision-making at multiple levels within 
an institution. Visualizations can also be a mechanism for increasing access to data. 
Some data concerning student demographics, student performance, and retention/ 
progression/completion can be found in national data sets. 


> Integrated Postsecondary Education Data System (IPEDS) 


> Educational Analytics at Center for Educational Effectiveness at University of 
California Davis 


> Curriculum flow and student progression 
» Learner profiles (e.g. “Know Your Students” / “Know Your Class Infographics”) 
>» Departmental Diagnostic Dashboard 


> Student Learning and Analytics at Michigan (SLAM) 


> Data Analytics to Study Student Pathways and Outcomes (DASSEE) at University of 
Colorado Boulder 


> Digital Innovation Greenhouse at the University of Michigan 


» Ecoach—Personalized Messaging to Students 


> Institute for Research on Innovation & Science 


» Pilot work to integrate information on all enrolled students with Census data 
open the possibility of comprehensive analysis and reporting on career outcomes 
for students at all levels. 


OBSERVATION PROTOCOLS: Observation protocols are especially relevant to 
understanding classroom behavior of instructors and the effects on student behavior 
and learning. 

> Classroom Observation Protocol for Undergraduate STEM (COPUS) 

> Generalized Observation & Reflection Protocol (GORP) 


> Observing Patterns of Adaptive Learning (OPAL) 


> Observation Protocol for Learning Environments (OPLE) 


> Reformed Teaching Observation Protocol (RTOP) 
> 3D Learning Observation Protocol (3D LOP) 


RUBRICS: Rubrics can help institutions, schools, departments, and instructors assess 
their own progress across a range of relevant areas. 


> Department Evaluation of Faculty Teaching Rubric at University of Kansas 


> Liberal Education and America’s Promise (LEAP) Initiative VALUE Rubrics 


> Partnership for Undergraduate Life Science (PULSE) Rubric 


FRAMEWORKS: Frameworks provide individual institutions and groups of institutions 
a shared model and common approach to advance a cycle of continuous and systemic 
improvement. 


> AAU Framework for Systemic Change in Undergraduate STEM Education 


> Bayview Alliance Driver Diagram 


> Increasing Student Success in STEM: A Guide to Systemic Institutional Change 


> Science Math Teaching Imperative (SMTI) Framework 


> StratEGIC Toolkit 


> Towards a Framework for Assessing and Promoting Teaching Quality at CU-Boulder 


STUDENT LEARNING ASSESSMENTS: Assessments of various kinds help to quantify 
student learning. 


> Concept inventories developed by disciplinary societies 
> Individual course assessments 


> Student Assessment of their Learning Gains 


SURVEYS: Surveys are useful to understand the attitudes, practices, opinions, and 
perceptions of both instructors and students. Institutions already use a range of 
surveys, both internal and external. Many also provide information that may aid in 
promoting change. 


> AAU Survey of Instructor Practices 


> Bay View Alliance Driver (BVA) Teaching Practice Survey 


> Consortium on Financing Higher Education (COFHE) 


> Describing Instructional Practice and Climate: Two New Instruments 


> Higher Education Research Institute (HERI) Freshman Survey 


> National Survey of Student Engagement (NSSE) 


> Project Ownership Survey 


> Student Experience in the Research University (SERU) Survey 


> Teaching Practices Inventory 


ADDRESSING CROSS-CUTTING CHALLENGES 


Efforts to assess the quality of undergraduate teaching and learning face several types 
of challenges, including the collection of data, variations in the ways universities have 
organized student data, as well as the rules and regulations on data governance, 
stewardship, sharing, and use. This section highlights several of the identified 
challenges and provides guidance for and examples of moving conversations about 
measuring teaching and learning forward. Ultimately, documenting institution-level 
effects of STEM education reforms requires finding “ways to defuse the potential 
conflict between locally useful classroom-level information and broader measures of 
program effects." 


> Each campus organizes data collection differently, including the location where data 
reside. Aggregating and joining data sets that are managed by different units within 
an institution may be challenging for technical, political, and institutional reasons. 
Particular types of information are also subject to different levels of restriction 
in terms of sharing and use (e.g., financial aid data versus academic performance 
data). Although the development of a single model for organizing data is unlikely, 
the partitioning of data sets in idiosyncratic ways is counterproductive to effective 
institutional decision-making, and makes cross-institutional comparisons much more 
difficult. Campuses have recognized this difficulty, and it would be advantageous for 
campuses to explore new ways to link data sets to support timely decision-making 
that benefits the institution while still protecting privacy. 


For example, the Student Data Matching Tool under development by the CREATE 
for STEM Institute at Michigan State University aggregates data and provides 

an interface for asking how a “treatment” (such as an undergraduate research 
experience) affects an outcome variable (such as graduating GPA) when students 
are matched on certain factors (e.g., Pell eligibility, race/ethnicity, gender). 
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Fairweather, J., Trapani, J., and Paulsen, K. (2015). The roles of data in promoting institutional 
commitment to undergraduate STEM reform: The AAU STEM Initiative experience. In Transforming 
institutions: 21st century undergraduate STEM education, ed. G. Weaver. West Lafayette, IN: Purdue 
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> The development of common data definitions, standards, formats, and 
methodologies to the extent possible by the community would greatly facilitate 
sharing, aggregation, and comparison of data. On many campuses there is a 
reluctance to adopt wholesale tools and techniques developed elsewhere. Custom- 
designed assessment tools can generate local buy-in for the purposes of educational 
reforms but can make even cross-college or department comparisons within a 
campus difficult. One goal of Essential Questions & Data Sources is to develop 
guiding principles to allow for meaningful sharing and comparisons within and 
across universities. Even agreeing on common file formats for data when using 
similar tools could be an important step forward. 


» BIG Academic Alliance 


» Association of American Universities Data Exchange (AAUDE) common data 
definitions 


» Common Education Data Standards (CEDS) developed by the U.S. Department 
of Education 


> Researchers and academic administrators should be clear about the kinds of analysis 
they seek to perform with institutional and student data, and distinguish research, 
evaluation, and assessment. Although these distinctions are sometimes nuanced, 
often they are more generic. Using some type of common definitions and formats 
can help institutions develop consistent guidelines for how to respond to different 
types of data requests. 


» One useful framework has been put together by the University of Wisconsin. 
It distinguishes academic research, institutional research, program evaluation, 
and student learning assessment based on criteria such as intent, funding 
source, performer, type of data used, and publication and dissemination of 
results. Such a framework can intersect with campus data governance models 
to help institutions become more systematic in how data are shared with 
researchers and academic administrators. 


> Institutional Review Board (IRB) roles and oversights differ among campuses. To 
the degree possible, campuses should find ways to allow IRBs to expedite review of 
studies that seek to improve educational performance using de-identified campus 
based student data for research, evaluation, and assessment purposes. 


> Much mythology has sprung up around FERPA, the Family Educational Rights and 
Privacy Act, which sets guidelines for protecting the privacy of student education 
records. FERPA is often invoked as a reason to prohibit sharing of information. 
Limitations in data sharing as the result of FERPA should be clarified and made 
consistent within, and to the extent possible between, institutions. Adhering to 
FERPA guidelines need not mean over-compliance. 


» In collaboration with the U.S. Census Bureau and the Universities of California, 
Michigan, and Texas, Institute for Research on Innovation & Science (IRIS) is 
conducting a pilot project to effectively link, rigorously analyze, and responsibly 
share data on student career outcomes and instruction that are derived from 
a variety of restricted administrative records. The membership FAQ and MOU 
address many questions related to data protection and sharing. 


> Researchers and academic administrators attempting to provide key information 
to institutional decision-makers are often unable to carry out this task because 
individual-level data are often separately housed on campus. De-identifying 
partitioned data sets would make within-institution (including cross-college and 
cross-department) analyses possible. Campuses might experiment with tactics for de- 
identification that preserve privacy while minimizing issues of campus jurisdictions in 
terms of questions that can be asked and answered with campus data. 


> The measurement of student learning and related outcomes to assess institutional 
level performance is rapidly evolving. Many metrics and measures are in a nascent 
stage. Institutions can help lead the charge on developing and sharing information. 
Sharing across institutions for internal benchmarking purposes is helpful to foster 
change. From the perspective of AAU, a higher education association that works 
with federal policymakers, aggregations of data across institutions can be useful for 
documenting impact, but institutional and personal anonymity must be maintained. 
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American Association for the Advancement of Science: Describing & Measuring 
Undergraduate STEM Teaching Practices 


Facilitating Innovation in Science Education through Assessment Reform 


Committee Discussion Document for the National Academy of Sciences Board 
on Higher Education & Workforce Quality in the Undergraduate Experience, 
December 4, 2015. 
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